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ABSTRACT 
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and its outcomes. This concept is illustrated through a discussion of 
evaluations of the Employment Intervention Demonstration Program, a 
multi -site research demonstration project funded by the Substance Abuse 
Mental Health Services Administration. At the Arizona site, a standardized 
interview protocol was used to gather quantitative information about 338 
participants and outcomes for them. A series of qualitative interviews was 
held with a smaller subset of participants. Combining the results of the two 
interview approaches gave a more complete picture of program effectiveness 
and characteristics. A crosswalk for the two data sets is described. (SLD) 
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Numbers & Words: Putting Them Together to Make Sense of a Program's Outcomes 

Michelle Stewart 
University of Arizona 

Background 

The Employment Intervention Demonstration Program (EIDP) is a multi-site research demonstration 
project funded by Substance Abuse Mental Health Services Administration in 1995. The Arizona site is 
examining the effectiveness of providing vocational services (supported employment) in conjunction with 
case management services using a multidisciplinary team approach to integrating services. Mental health 
consumers were randomly assigned to receive either the integrated teams or brokered vocational services 
from community service providers. To track the impact of the program in a uniform fashion, all sites are 
using a standardized interview protocol (the Common Protocol) to interview participants at entrance to 
the program and thereafter, every six months for two years. Key outcomes for the study include the 
number of people who return to work and job tenure, as well as satisfaction with the job and impact on 
life in general. 

In 1997, supplemental funding was obtained from the Social Security Administration with the purpose 
of conducting a series of qualitative interviews with a smaller subset of participants. These interviews 
focused on the individual’s decision to return to work, experiences with work, and in particular, 
experiences with receiving Social Security. These two interviews had separate purposes and asked 
different questions, but had some overlap. This paper is a brief overview of how they can be combined to 
get a fuller picture of a program and its outcomes. 

Samples 

The sample for the EIDP was drawn from two case management sites in Phoenix, Arizona. 
Participants had to have met the state’s definition of serious mental illness and have expressed an interest 
in working. A total of 338 baseline interviews were conducted with eligible participants. 

The subsample for the qualitative interview was drawn from the larger sample of participants; to be 
eligible, qualitative interviewees only had to have worked at some point during their tenure with the 
program. A total of 17 qualitative interviews have been completed to date, however, this paper reports on 
only three of these. The rest were deemed not useful because the interview was not audiotaped, the 
individuals had not worked more than a couple of days, did not receive Social Security benefits, or had 
not completed a quantitative interview while at the same job. Additional analyses are planned that will 
include the entire qualitative sample; the cases presented here are used for illustrative purposes to show 
how the two types of interviews could be used together. 

Methods 

Questions included in the Common Protocol were decided by the National Steering Committee 
for the project. Most of the Common Protocol is made up of well-known standardized instruments such as 
the Lehman Quality of Life (short version) that provide closed-ended response options for respondents to 
choose from. The interview is highly structured and quantitative in nature. In contrast, the qualitative 
questions were developed by the Arizona team as a standardized open-ended interview (Patton, 1990) and 
is more general in approach. For example, many of the qualitative questions ask respondents about their 
perceptions and feelings and allow them to respond in their own words. 

A cadre of experienced field interviewers well versed in standardized interviewing techniques 
administered the Common Protocol. A Ph.D. -level anthropologist trained as an ethnographer conducted 
the qualitative interviews. All the qualitative interviews presented here were audiotaped and transcribed; 
the transcription quality was spot-checked for accuracy. 
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Data Analysis & Results 

These two interviews were conducted for different purposes, with little forethought given to how they 
might be combined. Thus, this paper reflects a post hoc approach to combining the two approaches. In the 
nomenclature of Tashakkori and Teddlie (1998), this represents a “dominant-less dominant” study, in this 
case the dominant method is quantitative and the smaller component is the qualitative study. The goal was 
to bring the two interviews together in a complementary fashion (Greene, Caracelli, & Graham, 1989) 
with information gained from each method elaborating on that from the other. 

The first step to integrating these two interviews was to create a crosswalk between them listing what 
areas were covered by both and in what fashion they were covered (please refer to attached “Crosswalk”). 
As can be seen in the crosswalk table, there are considerable areas of overlap in the domains covered by 
quantitative questions taken from the Common Protocol and the standardized open-ended qualitative 
interview questions. Yet despite the apparent overlap at the level of the domains captured, it is clear that 
each interview is covering a different aspect of each domain. The qualitative interview covered many of 
these domains in greater detail than the quantitative interview. 

Once the crosswalk was completed, all the transcripts for the three qualitative interviews were read 
carefully, along with interviewer notes and case summaries where available. Using the domains of the 
crosswalk, the data from the qualitative and quantitative interviews were examined in iterative fashion. 
Using each domain as a framework, the results for each were summarized in narrative fashion. 

The domains of education, Social Security benefits, working, and social support are examined as 
examples. 

Education. 

The diagram below illustrates how the data from both sources were utilized to arrive at an integrative 
summary. From the quantitative interview, very little information can be gained. We know only the level 
of education they had attained, which allows us to use this variable in making predictions about who 
returns to work and what type of job they will get, but little else. From the qualitative data, a fuller picture 
is seen. Education is regarded by all 3 individuals as important, but only 2 more would like to return to 
school and neither of them has a clear idea about what type of education or training they would pursue. 
This is important in understanding outcomes of this project because the EIDP did not offer the 
opportunity to pursue further education or training, its emphasis is on placing participants into the 
workplace as soon as possible. Future programs may need to take into account that participants may 
prefer to return to school as a means of improving their employment level, but may need help narrowing 
down areas of possible study. 



Education 



• Two had some 
college and the other 
high school only 




• Want more education 

• Not sure what kind 

• Education is seen as 
important 




Two individuals felt that very strongly that more education was essential to 
helping them get a better job while the third one, who had attended two years 
of college, said although she feels education is important, she was not able to 
concentrate due to the voices and thus was not interested in pursuing further 
education. The male, who is working as a job coach in a vocational program, 
wanted education related to “human nature”. While the female was not sure 
what she would study, she was certain it would not be computers. 
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Social Security Benefits 

All three individuals had received Social Security Administration (SSA) benefits at some time, but 
only two were currently receiving them. One noted that she had not yet seen a reduction in her benefits 
due to working, but that she expected she would soon. Another said her mother is her payee and she is not 
aware of any reductions, though she thought her mother was saving her checks in case she had to pay any 
money back. The male celebrated going off SSA one year ago with his co-workers; the amount he got 
was gradually reduced over a period of time before that. One complained that SSA sent threatening letters 
and that she felt intimidated by them. All were grateful to be receiving the benefits to help them live, but 
one woman did note that sometimes it makes her feel “like a thief and adds to her feelings of 
worthlessness.” 

All three had held various jobs since beginning to receive benefits. Some of these jobs were “off the 
books” to avoid penalties or loss of their benefits yet enabling them to make some extra cash. One of the 
women currently receiving Social Security benefits reports receiving part of her salary in cash for that 
reason. Two of them felt that there needed to be someone at the case management site to support them in 
applying for benefits and for dealing with the paperwork required if someone returns to work. The 3 had 
very little knowledge regarding work incentive programs offered by the SSA; most had heard of the trial 
work period, but little else. None had ever applied for a PASS or IRWE. 



Social Security benefits 



• One person was 
no longer receiving SSA 
benefits; the other two 
got either SSI or SSDI 




• Few problems with it 

• Mixed feelings about 
receiving disability 

• Have worked on SSA 




All 3 had gone had applied for disability benefits during a stressful period 
(crisis) in their lives and were on it due to their mental illness. They expressed 
gratitude that they were able to receive them; one woman noted that she had 
mixed feelings commenting that although she is glad it is there, sometimes it 
makes her feels “like a thief and it adds to my feelings of worthlessness.” All 
had had various jobs while receiving benefits, both on and off the books. The 
male had gone off benefits a year ago because he found full-time employment 
with benefits; he and his coworkers had a celebration. All reported having 
received letters from the SSA threatening to terminate their benefits; two of 
the individuals were not distressed by the letters because they had someone to 
help them navigate the system that they could rely on. 



Working 

The key outcome for the EIDP is returning participants to the workplace; the expectation is that 
returning to work will improve the person’s life and will not result in exacerbation of symptoms. This is 
largely confirmed for these 3 individuals. Based on their comments during the qualitative interview, they 
indicate that working has been therapeutic for them - by taking their minds off their own problems and in 
one case, enabling the individual to learn to manage symptoms better. All agree that they are better off 
financially; their earned income is a substantial portion of their total monthly income. 
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The decision to return to work was largely due to desire to earn more money, but actually returning to 
the workplace has raised some anxieties as well. In particular, all 3 commented they had fears about not 
being seen as competent, or not being able to handle working again. One woman reported experiencing 
panic attacks initially after returning to work. Interestingly, in the quantitative interview, all agreed they 
were “very satisfied” (6 on a 7-point scale), but in the interview, one woman said she would be happier in 
a job where she didn’t have to deal with customers face-to-face. Similarly, disclosure of disability to the 
employer is viewed with mixed feelings. In one case, the woman had decided not to disclose for fear of 
how her employer might react. In the other two cases, they had disclosed before taking the job and were 
receiving reasonable accommodations to help them succeed. The reasonable accommodations concerned 
schedule changes and time off for doctor’s appointments. The male reported he wrote down a list of 
desired accommodations when he was offered the job; all were granted and he is in the process of 
obtaining formal ADA recognition from his employer by having his psychiatrist document his disability. 
Yet all 3 expressed lingering concerns over their abilities and competencies. Stigma in the workplace was 
encountered to a degree. One woman reported feeling she was hired as “damaged goods” and that she was 
“unwanted” at work; the male reported that although he works in a very supportive environment (the 
EIDP vocational program itself), he still catches fellow staff members making wisecracks about 
consumers and occasionally confronts them on it. 

An important point emerged regarding the importance of the psychiatrist in supporting people going 
back to work. All 3 reported their doctor asked about work and agreed that having the doctor inquire 
about work was important was important to them. One woman said she couldn’t believe her doctor could 
remember that she was working. 

Both of the people who are satisfied felt their jobs gave them a chance to help other people and that 
was important to them. Two people would rather work full-time, but currently only one is doing so. All 3 
noted that at times it is difficult to get out of bed and go to work. Their advice to vocational staff was to 
be patient with consumers who are struggling with returning to the workplace. Their advice for other 
consumers who want to go to work was “go ahead and do it”. 

Social Support 

The quantitative interview includes two subscales (total of 7 questions) pertaining to social relations in 
the Quality of Life (QOL) section. One subscale asks respondents to rate how satisfied they are with their 
social relations and the other is an objective rating of the frequency with which they have contact with a 
friend (or else) who is not a family member. One woman who reported living in a homeless shelter at the 
time she began her job, also reported having no contact with family and very few friends. As expected, 
her scores on both the QOL subscales were the lowest: she is the least satisfied with her social life and has 
the fewest social contacts. In contrast, the male respondent reported he is mostly satisfied with his social 
life while the final female rated her social life as “mixed” (between mostly satisfied and mostly 
dissatisfied). The male depends primarily on his wife and the woman on her mother. In both these cases, 
they reported experiencing social interactions on a daily basis. The figure below illustrates how these two 
were synthesized. 
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Social Support 







• On average they report 
feeling mostly satisfied 
with their social life 

• Two report nearly daily 
contact with nonfamily 




• Only two have people 
who support them 

• Both are satisfied with 
amount of support; third 
sometimes wishes for it 




The male felt his wife was his greatest supporter and she helps him in a 
variety of ways on a daily basis. One woman indicated her mother is her 
primary support at present. Both felt that they were receiving as much help as 
they needed. The third individual had few friends and no one she could really 
count on to help her regularly; she relies on the MH system for most of her 
supports and notes that although she feels she needs more, it is her choice. In 
both the cases where there is someone, that person was supportive of the 
return to work and instrumental in making it happen. 
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Conclusions 

This synthesis of the qualitative and quantitative interviews gave a fuller picture of the program and its 

outcomes. Several points are worth highlighting: 

> As hypothesized, participants report satisfaction with their jobs and view work as therapeutic, though 
the qualitative data gives a much fuller picture of how work has actually helped them cope with their 
mental illness 

> Why do they work: money, feel useful, have a routine, love what they do. As one said “I thought if I 
was working, I could get my life together.” 

> Work has not lead to an exacerbation of symptoms, although all reported an increase in anxiety 
regarding work performance 

> As seen in the larger quantitative dataset, having held a job in the 5 years prior to joining the program 
is useful to predict whether or not someone will get a job 

> The role of the psychiatrist as a member of the multidisciplinary team is far more important in 
promoting work than previously realized 

> Social support - particularly having someone encouraging the return to work - appears to play an 
important role 

> Dealing with the Social Security Administration can be anxiety-provoking, but this can be lessened 
by having someone who is knowledgeable and supportive help participants navigate the system 

> Although the SSA benefits helped them survive, there is some shame and guilt involved in receiving 
disability benefits and they would prefer to be off them and able to earn their own way 



Caveats & Advice 

Although this effort to combine the qualitative and quantitative interviews gave added insight, it is not 

without problems: 

> It proved very difficult to combine two interviews designed for different purposes - if you are going 
to use mixed methods, plan for it from the beginning and prepare both the research design and 
analyses plans accordingly 

> Requires an iterative process and was time-consuming 

> The use of open-ended questions with persons with serious mental illness can be somewhat 
problematic and the interviewer needs to be experienced in conducting qualitative interviews and 
have some exposure to mental illness 
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Domain 


Quantitative from Common Protocol 


Qualitative from Standardized Open- 
ended interview 


Personal history 


> Date of birth 


V Date of Birth 

> Where born 

V Where grew up 

> How long in AZ 


Ethnicity/culture 


> Ethnicity - categorical variable 


> Ethnicity 

V Number of languages speak 


Education 


> Years of education - categorical 


V Years of education 

V Do you want more education 

V What kind of education 

V How important is more education to 
get a better job 


Social Security 
benefits 


> SSA myth questions - Likert scale 

> Amount of income from SSA 
benefits 


y Receive yes/no 

> Have you had problems with it? 

> Were you told you could not work 

again? | 

> How did you feel about that 
y Knowledge of work incentive 

programs 

V Communication with SSA office 

> Primary source of information on it 

> Best way for SSA to get info to MH 
consumers 

y How would you rate them? 
y What one thing would you tell SSA 
'y Feelings about receiving disability 
payments 

> Work history since receiving benefits 

> Impact of work on SSA 

> Problems specifically related to that 

> Maximum hours can work and keep 
benefits 


Working 


> Amount of earned income 

> Work motivation scale - Likert scale 

> Disclosure of disability to employer 
- yes/no 

> Job satisfaction - Likert scale rating 

> Life better/worse - categorical 


y What prompted you to go back to 
work 

> Childcare needs 

> Would you like to work more hours 

> How did you get the job 

> How do you feel about the job 

> What makes you feel like you don’t 
want to go to work 

y What is perfect/ideal job 
y Would you rather work full or part- 
time 

y Disclosure - who/how/concerns about 
it 

> Reasonable accommodations 

y Discrimination/stigma on the job 
y Experiences with sexual harassment 
y Psychiatrist's knowledge of job 

> Advice for other consumers who want 
to work 
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Vocational 

services 


> Satisfaction (Hargreaves) 
questionnaire - Likert scale 


> Satisfaction 

> What is most useful about services 

> Advice for vocational staff 


Income 


> Monthly income 

> Satisfaction with income (Lehmann 
Quality of Life) - Likert scale 

> Amount of money to spend on self 
(luxuries) 


> Monthly income 

> Enough to pay bills 

> How are you making it 

> Money for luxuries 

> Is it financially necessary for you to 
work 


Social Support 


> A few questions on social support 
from the Quality of Life - Likert 
scale 


> Who encouraged you to go to work 

> Social support questions - who helps 
you 

> What is relationship with one person 
who helps you most 

> How has that relationship changed 
over time 

> Satisfaction with social support 
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